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Abstract 

The processing of mega-dimensional data, such as images, scales lin- 
early with image size only if fixed size processing windows are used. It 
would be very useful to be able to automate the process of sizing and inter- 
connecting the processing windows. A stochastic encoder that is an exten- 
sion of the standard Linde-Buzo-Gray vector quantiser, called a stochas- 
tic vector quantiser (SVQ), includes this required behaviour amongst its 
emergent properties, because it automatically splits the input space into 
statistically independent subspaces, which it then separately encodes. 

Various optimal SVQs have been obtained, both analytically and nu- 
merically. Analytic solutions which demonstrate how the input space is 
split into independent subspaces may be obtained when an SVQ is used 
to encode data that lives on a 2-torus (e.g. the superposition of a pair 
of uncorrelated sinusoids). Many numerical solutions have also been ob- 
tained, using both SVQs and chains of linked SVQs: (1) images of multi- 
ple independent targets (encoders for single targets emerge), (2) images of 
multiple correlated targets (various types of encoder for single and multi- 
ple targets emerge), (3) superpositions of various waveforms (encoders for 
the separate waveforms emerge - this is a type of independent component 
analysis (ICA)), (4) maternal and foetal ECGs (another example of ICA), 
(5) images of textures (orientation maps and dominance stripes emerge). 

Overall, SVQs exhibit a rich variety of self-organising behaviour, which 
effectively discovers the internal structure of the training data. This 
should have an immediate impact on "intelligent" computation, because 
it reduces the need for expert human intervention in the design of data 
processing algorithms. 



1 Stochastic Vector Quantiser 
1.1 Reference 

Luttrell S P, 1997, Mathematics of Neural Networks: Models, Algorithms and 
Applications, Kluwer, EUacott S W, Mason J C and Anderson I J (eds.), A 
theory of self-organising neural networks, 240-244. 

*This talk was given at the Workshop on Self-Organising Systems - Future Prospects for 
Computing, 28-29 October 1999, Manchester, UK. 
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1.2 Objective Function 

1.2.1 Mean Euclidean Distortion 

D= j rfa; Pr(x)^Pr(y|x) j dx' Vv {x' \y) \\x - x' f (1) 
y 

• Encode then decode: x — > y — > x'. 

• X ~ input vector; y = code; x' = reconstructed vector. 

• Code vector y = (yi, ?/2, • • ■ , Vn) for 1 < < M. 

• Pr (x) = input PDF; Pr {y\x) = stochastic encoder; Pr {x' \y) = stochas- 
tic decoder. 

• ||x — x'll^ = EucUdean reconstruction error. 

1.2.2 Simplify 

D = 2 dxPv{x)^Fi:{y\x)\\x-x'{y)f 

•' y 

, , , /" , ^ / , ^ f da; Pr (a;) Pr (y la;) a; 

x' iy) = / Pr(x k/)x = \\ — \ , (2) 

J ^ '"^ /da; Pr(a;)Pr(y |a;) ^' 

• Do the / da; Pr (a; |y ) (• • • ) integration. 

• x' {y) = reconstruction vector. 

• x' {y) is the solution of = 0, so it can be deduced by optimisation. 

1.2.3 Constreiin 

Pr(y|a;) = Pr {yi\x)PT {y2\x) ■ ■ ■ Pr {yn\x) 
1 " 

1=1 

• Pr(y|a;) implies the components (yi,j/2, •■• ,yn) of y are conditionally 

independent given x. 

• x' {y) implies the reconstruction is a superposition of contributions x' {yi) 
for i = 1, 2, • • • , n. 

• The stochastic encoder samples n times from the same Pr (y |a;). 
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1.2.4 Upper Bound 



D < D1+D2 



2 r ^ 

£)i = - dxPrix)J2P^{y\x)\\x-x'{y)\\ 



Do 



2 (n - 1) 



J dx Pr(a;) 



M 



X -^PT{y\x)x' (y) 



(4) 



• £>! is a stochastic vector quantiser with the vector code y replaced by a 
scalar code y. 

• D2 is a non-linear (note Pr(y|a;)) encoder with a superposition term 
E^iPr {y\x)x' {y). 

• n — > 00: the stochastic encoder measures Pr{y\x) accurately and D2 
dominates. 



• n — > 1: the stochastic encoder samples Pr(y|a;) poorly and £>i domi- 
nates. 



2 Analytic Optimisation 
2.1 References 

Luttrell S P, 1999, Combining Artificial Neural Nets: Ensemble and Modu- 
lar Multi-Net Systems, 235-263, Springer- Verlag, Sharkey A J C (ed.), Self- 
organised modular neural networks for encoding data. 



Luttrell S P, 1999, An Adaptive Network For Encoding Data Using Piecewise 
Linear Functions, Proceedings of the 9th International Conference on Artificial 
Neural Networks (ICANN99), Edinburgh, 7-10 September 1999, 198-203. 



2.2 Stationarity Conditions 
2.2.1 Stationarity w.r.t. x' (y) 



n J dxPr{x\y)x = x' {y) + {n-l) J dx Pr (a; |j/ ) ^ Pr {y' \x) x' {y') (5) 

j/'=i 

• Stationarity condition is = 0. 
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2.2.2 Stationarity w.r.t. Pr(t/'|x) 



M 



Pr (a;) Pr {y\x)J2 (P^ (j/' \x ) - (5,,,0 x' {y') .\\x' (y') - n x + (n - 1) ^ Pr {y" \x ) x' {y") I = 



M 



y'=i 



3/"=l 

(6) 

• Stationarity condition is qx^^I^\1) = 0, subject to X^^i Pr {y\x) = 1. 

• 3 types of solution: Pr (a;) = (trivial), Pr (y |a; ) = (ensures Pr {y\x) > 
0),and E^=i {Pviy'\x)-Sy,y,)i---)=0. 

2.3 Circle 

2.3.1 Input vector uniformly distributed on a circle 

X = (cos 9, sin 9) 

r2-K 



j dx¥v{x){-..) = l-jjd9{...) 



(7) 



2.3.2 Stochastic encoder PDFs symmetrically arramged around the 
circle 



Pr(y|0)=p(0-^ 



(8) 



2.3.3 Reconstruction vectors symmetrically arraged around the cir- 
cle 



(9) 



2.3.4 Stochastic encoder PDFs overlap no more than 2 at a time 

'l O<|0|<^-,s 



p{o) = {no) ii-s<\e\<j^ + s 

\0\>^t+s 



(10) 
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2.3.5 Stochastic encoder PDFs overlap no more than 3 at a time 

7iW o<m<-^+s 

(11) 



/2 (0) 



■s<\0\< 



^' ^ h{0) '^-s<\e\<^ + s 

\0\>ii+s 




2.4 2- Torus 

2.4.1 Input vector uniformly distributed on a 2-torus 



X 
X2 



{XI,X2) 

(cos ^1, sin ^i) 
(cos ^2, sin 62) 



J dxPv{x){---) = dOiJ^ de2{---) 

2.4.2 Joint encoding 

Pr {y\x) = Pr(y |a;i,a;2) 

• Pr {y \x ) depends jointly on x\ and X2- 

• Requires n = 1 to encode x. 

• For a given resolution the size of the codebook increases exponentially 
with input dimension. 



c2ir 



(12) 



(13) 
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2.4.3 Factorial encoding 

• Yi and Y2 arc non-intersecting subsets of the allowed values of y. 

• Pr {y \x ) depends either on Xi or on X2, but not on both at the same time. 

• Requires n » 1 to encode x. 

• For a given resolution the size of the codebook increases linearly with 
input dimension. 



2.4.4 Stability diagram 

• Fixed n, increasing M: joint encoding is eventually favoured because the 
size of the codebook is eventually large enough. 

• Fixed M, increase n: factorial encoding is eventually favoured because the 
number of samples is eventually large enough. 

• Factorial encoding is encouraged by using a small codebook and sampling 
a large number of times. 




3 Numerical Optimisation 

3.1 References 

Luttrell S P, 1997, to appear in Proceedings of the Conference on Information 
Theory and the Brain, Newquay, 20-21 September 1996, The emergence of dom- 
inance stripes and orientation maps in a network of firing neurons. 

Luttrell S P, 1997, Mathematics of Neural Networks: Models, Algorithms and 
Applications, Kluwer, Ellacott S W, Mason J C and Anderson I J (eds.), A 
theory of self-organising neural networks, 240-244. 

Luttrell S P, 1999, submitted to a special issue of IEEE Trans. Information 
Theory on Information- Theoretic Imaging, Stochastic vector quantisers. 
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3.2 Gradient Descent 

3.2.1 Posterior probability with infinite range neighbourhood 

J2y'=iQix\y) 

• Q {x\y) >0 is needed to ensure a valid Pr (y |a;). 

• This does not restrict Pr (y |a;) in any way. 

3.2.2 Posterior probability with finite range neighbourhood 

(16) 

• {y') is the set of neurons that lie in a predefined "neighbourhood" of y' . 

• {y) is the "inverse neighbourhood" of y defined as A/""^ (y) = {y' : y GAf {y')}- 

• Neighbourhood is used to introduce "lateral inhibition" between the firing 
neurons. 

• This restricts Pr (y but allows limited range lateral interactions to be 
used. 

3.2.3 Probability leakage 

Pr(y|a;)^ ^ Pr (y |y' ) Pr (y' ) (17) 

y'G£-i(y) 

• Pr (y |y' ) is the amount of probability that leaks from location y' to loca- 
tion y. 

• C iy') is the "leakage neighbourhood" of y'. 

• (y) is the "inverse leakage neighbourhood" of y defined as (y) = 
{y':ye/:(y')}- 

• Leakage is to allow the network output to be "damaged" in a controlled 
way. 

• When the network is optimised it automatically becomes robust with re- 
spect to such damage. 
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• Leakage leads to topographic ordering according to the defined neighbour- 
hood. 

• This restricts Pi{y\x), but allows topographic ordering to be obtained, 
and is faster to train. 

3.2.4 Shorthand notation 

Lyy=Pr{y\y') Py.y = Ft (y \x;y' ) 

dy = x - x' (y) {L d)y = Y.y'ec{y) ^y',v ^v' 

{PLd)^ ^ T.y>emy)Py,y' {Ld)^, (P^PLd)^ ^ Ey^.^-^y) Py',y (P Ld)^, 

ey = \\x-x'{y)f {Le)y = J2y'(^ciy) ^y,y' ^y' 

{PLe\ ^ T.y-emy) Py,y' iLe\, (P^PLe) ^ E,'eiv-(,) Py',y {PLe)y, 

d ^ Er=i {L^P)ydy OTd ^ E^i {PLd\ 



(18) 



• This shorthand notation simplifies the appearance of the gradients of Di 
and £>2- 

• For instance, Pr(t/|cc) = [L'^p)^. 
3.2.5 Derivatives w.r.t. x' {y) 

dDi 4 



j dxVT{x) {L^p)ydy 
x'iy) - nM^ j dxVrix){L-p)^-d (19) 



x' {y) nM 
9£>2 4 (n - 1) 



• The extra factor jj in arises because there is a J2y=i i' ") hidden 

inside the d. 

3.2.6 Functional derivatives w.r.t. logQ{x\y) 

= dxPv{x)Y,5logQ{x\y)(py{Le)^-{P^PLe)^) (20) 

j/=i 

- ^""^ > dxPr{x)J2SlogQ{x\y)(py{Ld)^-{P^PLd)^).d 

■' y=i 



Differentiate w.r.t. logQ {x \y) because Q {x\y) > 0. 
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2.7 Neural response model 

Q{x\y) 



1 



I + Qxj) {-w (y) .X - b (y)) 
• This is a standard ''sigmoid" function. 



(21) 



• This restricts PY{y\x), but it is easy to implement, and leads to results 
similar to the ideal analytic results. 

2.8 Derivatives w.r.t. w {y) and h {y) 

dDi 2 



Hy) 

w{y) 
dD2 



h{y) 

w{y) 



j dxVv{x){py{Le\-{P'^PLe)^){l-Q{x\y))(^l 
^^^^ I dxV,{x)(py{Ld\-{P^PLd)^) .d{l-Q{x\y)) 



.3 Circle 

3.1 Training history 



/ ' 

= ■ / \> 
H^^^^""^ ^ 




1 ■ 


" \ 

\ j 
\ / 

! / 



-1 -0.5 □ 



• M = 4 and n = 10 were used. 

• The reference vectors x' (y) (for y = 1,2,3,4) are initialised close to the 
origin. 

• The training history leads to stationary x' {y) just outside the unit circle. 
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3.3.2 Posterior probabilities 




-2-10 1 2-2-10 12 




-2-10 1 2-2-10 12 



• Each of the posterior probabihties PT{y\x) (for y = 1,2,3,4) is large 
mainly in a ^ radian arc of the circle. 

• There is some overlap between the Pr (y\x). 
3.4 2-Torus 

3.4.1 Posterior probabilities: joint encoding 




0123456 01234 5 6 0123456 0123456 




0123456 0123456 0123456 0123456 



• M = 8 and n — 5 were used, which lies inside the joint encoding region of 
the stability diagram. 

• Each of the posterior probabilities Pr {y\x ) is large mainly in a localised 
region of the torus. 

• There is some overlap between the Pr (y |a; ). 
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3.4.2 Posterior probabilities: factorial encoding 




□ 123456 0123456 0123456 0123456 



■ 



□ 1 2 3 4 S 6 




234S6 0123456 



• Af = 8 and n ~ 20 were used, which lies inside the factorial encoding 
region of the stability diagram. 

• Each of the posterior probabilities Pr(y|a;) is large mainly in a collar- 
shaped region of the torus; half circle one way round the torus, and half 
the other way. 

• There is some overlap between the Pr (y |a; ) that circle the same way round 
the torus. 

• There is a localised region of overlap between a pair of Pr (y |a; ) that circle 
the opposite way round the torus. 

• These localised overlap regions are the mechanism by which factorial en- 
coding has a small reconstruction distortion. 

3.5 Multiple Independent Targets 
3.5.1 Training data 




• The targets were unit height Gaussian bumps with a = 2. 

• The additive noise was uniformly distributed variables in [0,0.5]. 
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3.5.2 Factorial encoding 



• M = 10 and n = 10 were used. 

• Each of the reference vectors x' (y) becomes large in a localised region. 

• Each input vector causes a subset of the neurons to fire corresponding to 
locations of the targets. 

• This is a factorial encoder because each neuron responds to only a subspace 
of the input. 

3.6 Pair of Correlated Tcirgets 
3.6.1 Training data 




D S 10 15 2 

vector element 



• The targets were unit height Gaussian bumps with a = 1.5. 



13 



3.6.2 Training history: joint encoding 




• M = 16 and n = 3 were used. 



• Each of the reference vectors x' (y) becomes large in a pair of localised 
regions. 

• Each neuron responds to a small range of positions and separations of the 
pair of targets. 

• The neurons respond jointly to the position and separation of the targets. 
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3.6.3 Training history: factorial encoding 




• M = {16, 16} and n = {20, 20} were used; this is a 2-stage encoder. 

• The second encoder uses as input the posterior probability output by the 
first encoder. 

• The objective function is the sum of the separate encoder objective func- 
tions (with equal weighting given to each). 

• The presence of the second encoder affects the optimisation of the first 
encoder via "self-supervision". 

• Each of the reference vectors x' {y) becomes large in a single localised 
region. 
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3.6.4 Training history: invariant encoding 




• M = {16, 16} and n = {3, 3} were used; this is a 2-stage encoder. 

• During training the ratio of the weighting assigned to the first and second 
encoders is increased from 1:5 to 1:40. 

• Each of the reference vectors x' (y) becomes large in a single broad region. 

• Each neuron responds only the position (and not the separation) of the 
pair of targets. 

• The response of the neurons is invariant w.r.t. the separation of the 
targets. 
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3.7 Sepairating Different Waveforms 
3.7.1 Training data 




• This data is the superposition of a pair of waveforms plus noise. 

• In each training vector the relative phase of the two waveforms is randomly 
selected. 



3.7.2 Training history: factorial encoding 




• M = 10 and n = 20 were used. 

• Each of the reference vectors x' {y) becomes one or other of the two wave- 
forms, and has a definite phase. 

• Each neuron responds to only one of the waveforms, and then only when 
its phase is in a localised range. 



17 



3.8 Maternal + Foetal ECG 
3.8.1 Training data 











mm 































• This data is an S-cliannel ECG recording taken from a pregnant woman. 

• Tlie large spikes are the woman's heart beat. 

• The noise masks the foetus' heartbeat. 

• This data was whitened before training the neural network. 

3.8.2 Factorial Encoding 




• M = 16 and n = 20 were used. 

• The results shown are w{y).x computed for all neurons (y = 1, 2, • ■ • , 8) 

for each 8-dimensional input vector x. 

• After limited training some, but not all, of the neurons have converged. 

• The broadly separated spikes indicate a neuron that responds to the 

mother's heartbeat. 

• The closely separated spikes indicate a neuron that responds to the foetus' 
heartbeat. 
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3.9 Visual Cortex Network (VICON) 
3.9.1 Training Data 




• This is a Brodatz texture image, wliose spatial correlation length is 5-10 
pixels. 
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3.9.2 Orientation map 




mwmamvmxaaosMua^Xd^rjrim 

^;!/yi/L/.'f,i«irj'5'V!^,fi?sjjuEii£Kffi!3i 

afjy'iY¥W5L*Wf/f/'.JL[[i:[i:ai*fifJesS;«flfifip«**yS!RI 
Si»!l>llL£[£^££tSfiiaUll[<nirSfSCiaLI>;t£K;irj»A/jfil3a 




• M = 30 X 30 and n = 1 were used. 

• Input window size = 17 x 17, neighbourhood size = 9x9, leakage neigh- 
bourhood size = 3x3 were used. 

• Leakage probabiUty was sampled from a 2-dimensional Gaussian PDF, 
with (T = 1 in each direction. 

• Each of the reference vectors x' (y) typically looks like a small patch of 
image. 

• Leakage induces topographic ordering across the array of neurons 

• This makes the array of reference vectors look like an "orientation map". 

3.9.3 Sparse coding 




The trained network is used to encode and decode a typical input image. 
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• Left image = input. 

• Middle image = posterior probability. This shows "sparse coding" with a 
small number of "activity bubbles". 

• Right image — reconstruction. Apart from edge effects, this is a low 
resolution version of the input. 

3.9.4 Dominance stripes 




• Interdigitate a pair of training images, so that one occupies on the black 
squares, and the other the white squares, of a "chess board". 

• Preproccss this interdigitated image to locally normalise it using a finite 

range neighbourhood. 

• M = 100 X 100 and n = 1 were used. 

• Input window size = 3x3, neighbourhood size = 5x5, leakage neigh- 
bourhood size = 3x3 were used. 

• Leakage probability was sampled from a 2-dimensional Gaussian PDF, 
with (T = 1 in each direction. 

• The dominance stripe map records for each neuron which of the 2 inter- 
digitated images causes it to respond more strongly. 

• The dominance stripes tend to run perpendicularly into the boundaries, 
because the neighbourhood window is truncated at the edge of the array. 
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